In enterprise computer and networking systems, it is often advantageous to utilize high-density computer processing, data storage and telecommunication hardware components for the processing, storage, switching, routing and transport of high speed data in the form of digital signals. It is also advantageous for a plurality of these various components to communicate with each other at very high speed signaling rates. The use of a component-based system having separation of functions such as processing, storage, switching, and input/output interfaces allows individual components to be updated or upgraded independently from other components as well as allows customization for specific tasks. Furthermore, the use of components is cost effective since developing or purchasing a new component is less expensive than developing or purchasing an entirely new integrated hardware system that is not based on component design. However, such existing systems are hardwired and use a central switching architecture to allow components to communicate with one another.
Furthermore, the modern data center is suffering from the constraints of cabled, fixed-lane architectures. The concept of interconnecting racks of computing/storage servers through switching equipment with high-speed cables/fibers is taken for granted as the only method for providing connectivity. The fundamental technique used to provide random server-to-server communications is dominated by the Ethernet switch, and in modern data centers the switching architecture is typically implemented through a tiered tree design. A typical equipment rack contains 20 to 40 servers that connect with Ethernet cables to the top-of-rack (TOR) switch. The TOR switches are then interconnected to the next tier in the switching system to an end-of-rack (EOR) switch (also known as a cluster switch). The EOR switch is normally a 10 gigabit Ethernet (10 GigE) switch with 100's of ports. The EOR/cluster switches are then interconnected with 10 GigE (heading towards 40 GigE in the near future) uplinks to the next tier. A diagram of this type of system often resembles a tree, and the interconnecting technologies are referred to as fat trees indicating higher-bandwidth interconnections near the “root” in an attempt to provide maximal non-blocking connectivity. The tree architecture requires that the switching be connected through several layers or stages in order to implement the required connectivity. Intra-rack connections are usually 1 GigE to the TOR switch, and the uplink connections are 10 GigE to the cluster switch, although higher speed connections are anticipated in the future heading towards 40 GigE and 100 GigE rates.
The implied switching architecture for the tree, or any system that uses fixed cabling for interconnection, is that all data must traverse a common or central switching system. Like the traditional central office with circuit switching that is centralized, the tree also has central elements at the higher tiers in the structure. This is simply a necessary fact for physically connected (cabled) systems. Like a highway with fixed on and off ramps, a car must use the lanes and ramps to get from point A to point B—there is no facility for randomly traversing two locations other than the infrastructure of dedicated lanes and ramps. The fixed cabling in the data center is its own constraint—the industry has built these centers based on this architectural assumption.
The efficiency of the networking architecture for a data center is affected by switch bandwidth, power, area required (including the impact of cabling infrastructure), and total system capacity. The current switching topologies are constrained by the maximum data communications bit rate that can be carried over cable or fiber as well as the practical limit on the number of switch input and output ports. Given these constraints, the switching architectures have evolved from the full non-blocking matrix to tree structures. By creating a tree with tiered levels, the maximum number ports per switch element can be limited to allow implementation using commodity switching integrated circuits.
As the size of the data center grows upwards towards 100,000 servers, the number of tiers in the switching topology in turn must grow in order to control footprint and costs. An example of a proposed switching architecture for a large data center may include a middle tier (ingress/egress switches or aggregation level) and two upper tiers of the topology having a total of 216 10-gigabit (10 GigE) switches, each with 144 ports, in addition to 5184 Top Of Rack (TOR) switches designed to support a total of 103,680 servers. Below the middle tier are the racks containing the servers. Implied in this design are 20 servers per rack. Each rack contains a TOR switch with twenty (20) 1 GigE ports for the servers and two (2) 10 GigE uplink ports for connectivity to the next tier. The upper tier (intermediate node/core switches) carries the highest bandwidth traffic. Since the bandwidth at the root is higher than the leaves (at the servers), the tree is designated as a fat tree.
In this example, the power required for each 10 GigE switch is approximately 15 kW, totaling to over 3 MW for the middle and upper tiers. The number of 10 GigE uplink cables from the TOR switches is 5184×2=10,368. The number of cables between the second and third tier is also 10,368, totaling to 20,736 10 GigE cables for the system. The full bisection bandwidth of this proposed fat tree topology is 103.68 Tb/s. Switching latency for a multi-tiered topology is incurred by traversing each stage of the switched path. In this example data leaves the source TOR switch and must move up through the second and third tiers and then back down to the destination TOR switch, accumulating 3 stages of switch latency plus the propagation delay in the cables. Assuming fiber cabling for all the 10 GigE connections, and using the fastest known 10 GigE switching equipment, the total (layer-1 only) latency comes out to a minimum of 2 μs.
Other approaches have proposed the reduction of switch equipment costs of a fat tree using only 1 GigE switches in order to leverage the lower costs of commodity level GigE products. In this example, the fat tree is implemented using 2880 commodity 48-port GigE switches providing 27.648 Tb/s of bisection bandwidth (hosting 27,638 servers). However, the savings in switch equipment costs would be obscured by the massive cabling required to interconnect the switches. It is estimated that that over 200,000 meters of cable would be required weighing nearly 10,000 kg or 22,000 pounds. This level of cabling complexity would significantly impact the design and cost of the proposed data center infrastructure.
What is needed is a system and method for high speed signaling in a backplane fabric that is not limited by fixed physical media and/or a centralized switching architecture.